1. Field of the Invention
The present invention is directed to distributed network applications in a clustered computing environment, and more particularly, to methods and apparatus for processing administrative requests of the distributed application in such an environment.
2. Description of the Prior Art
Distributed network applications often have the concept of a master machine that performs administration for the entire distributed application. In this case, administration includes bringing a component online, taking a component offline, or checking the status of an individual component. Software architectures that permit multiple computers to operate together in a cluster to improve overall availability, such as the architecture of Microsoft® Cluster Server (MSCS), do not fit the centralized administration model of such distributed network applications. With MSCS, for example, a custom resource DLL on each node of a cluster must implement administrative operations for that node, such as bringing a component online, taking a component offline, or checking the status of a component. Thus, when an administrative operation (e.g. offline, online, etc.) is to be performed on a given node, MSCS invokes the resource DLL on that node to perform the operation. This is contrary to the centralized administration model of many distributed network applications which requires that all such operations be initiated and controlled by a single, master node in the cluster—i.e. if a node is to be brought online or offline, for example, that operation would be initiated and performed by the master node for the distributed application. As a more specific example, this problem arises, for example, when a distributed application written for BEA Systems, Inc.'s Tuxedo® transaction manager and messaging middleware is executed in a clustered computing environment under the control of Microsoft's Cluster Server (MSCS) software.
A. Tuxedo
In a Tuxedo environment, one or more Logical Machines can be grouped together to define a Tuxedo Domain. A Logical Machine represents a Tuxedo server machine and often corresponds to a physical computer system (i.e., one node of a network), but this is not always the case—in some cases, more than one Logical Machine can be defined on a single node.
Each Logical Machine contains Tuxedo Admin Servers, including a BBL process, a Bridge process, and possibly a DBBL process. Each Logical Machine also has a Tuxedo TListen process. The BBL process monitors and controls user-written Servers. The Bridge process on one Logical Machine connects to the TListen process on another Logical Machine. One of the Logical Machines in the Domain is designated as the Master; the Master may also have a Backup configured. The DBBL is run on the Logical Machine which is acting as the Master. The DBBL coordinates and controls the various BBLs, i.e., it performs administration for the Domain, including bringing a component online, taking a component offline, or checking the status of an individual component.
The user-written Servers on a Logical Machine are grouped into one or more Tuxedo Server Groups. If the Tuxedo Server Group supports transactions, transaction states are logged to a TLOG. Each Logical Machine has its own distinct TLOG. Other state and configuration information is stored in a binary TUXCONFIG file. Each Logical Machine has its own copy of the TUXCONFIG, but the DBBL keeps the various TUXCONFIG files synchronized.
B. Microsoft Cluster Server (MSCS)
In the current release of Microsoft® (Windows NT®, Enterprise Edition, a cluster refers to two connected systems (usually called nodes) that act as a highly available system. These two systems have one or more physical disk drives on a shared SCSI bus; each disk on the shared SCSI bus may be accessed by either system, but only by one system at a time. If one of the systems in the cluster fails, the other system gains control of the shared disk(s) and continues to provide service, picking up where the failed system left off. Future releases of Windows NT are expected to support clusters with more than two nodes.
MSCS is controlled by the Cluster Service which runs on each node of the cluster. The Cluster Service spawns one or more Resource Monitors. A Resource Monitor watches over one or more resources. A resource is any entity which can be monitored and controlled by MSCS. The Resource Monitor calls entry points in a Resource DLL (as defined in the MSCS Resource API) to monitor and control a particular resource. In particular, a Resource Monitor calls entry points in the Resource DLL at appropriate times to check the state of the resource, bring the resource online, or take the resource offline. Note, therefore, that the Resource DLL implements the action needed to bring the resource online or take the resource offline—administrative operations that in a Tuxedo environment must be performed by the Tuxedo master. Moving a resource from one node to the other occurs by taking the resource offline on the first node and bringing the resource online on the second node.
MSCS includes a Resource DLL (clusres.dll) that defines several resource types. Below is a list of relevant resource types supported by clusres.dll:
Physical Disk—controls a physical disk drive located on the shared SCSI bus.
IP Address—defines an IP address which can be dynamically allocated to one node or the other.
Network Name—defines a symbolic name for an IP address resource.
Generic Service—controls any Windows NT service.
Generic Application—controls well-behaved Windows NT console application programs.
In addition, third party developers can create custom resource types by developing and registering a Resource DLL that conforms to the Resource API.
MSCS resources can be grouped together into a Resource Group. A Resource Group is the basic unit of failover; that is, every resource in a particular group runs on the same node at a given point in time. The resources within a Resource Group can have dependencies that control the relative order of online and offline operations.
Typically, a Resource Group will contain one or more Physical Disk resources, an IP Address resource, a Network Name resource, and one or more additional resources representing a server application, such as Generic Service resources, Generic Application resources, and/or custom resource types. A Resource Group that has its own IP Address resource and Network Name resource is known as a Virtual Server.
A Virtual Server appears to an external client running a TCP/IP client/server type application as a distinctive server computer. In reality, there may be several Virtual Servers running on a single node of an MSCS cluster, each with different IP addresses. Furthermore, the Virtual Server can move from one node of the MSCS cluster to the other, and this is transparent to the client (except for a momentary interruption or slow down in service).
FIG. 1 illustrates an exemplary cluster comprising two nodes 10, 12 each running a respective virtual server 14, 16. A Cluster Service 18 running on each node controls the cluster, including, for example, initiating administrative requests to the Resource Monitors (not shown) of each virtual server 14, 16. Each node 10, 12 is connected to a plurality of disks 20 via a shared bus 24, such as, for example, a SCSI bus. Each node 10, 12 is also connected via a respective network interface to a local area network (LAN) 22.
Referring now to FIG. 2, if the second node 12 of the cluster fails, MSCS starts the failed Virtual Server 16 on the first node 10 of the cluster. This is a failover. At this point, both Virtual Servers 14, 16 are running on the first node. Client programs continue to run normally (with the exception of possibly degraded performance).
When the second node 12 of the cluster resumes normal operation, MSCS takes the failed-over Virtual Server 16 offline on the first node 10. Then MSCS brings this Virtual Server 16 back online on the second node 12. This is a failback. At this point, the configuration shown in FIG. 1 has resumed, in which each node is running one Virtual Server.
C. Deploying a Distributed Network Application in a Clustered Environment
A distributed network application, such as an application designed to run in the Tuxedo environment, can be set-up to run in a clustered computing environment, such as MSCS. For example, referring to FIG. 1, a Tuxedo Domain with two Logical Machines could be configured with the two MSCS Virtual Servers 14, 16. During normal operations, one Virtual Server (and therefore one Logical Machine) runs on each of the nodes 10, 12 of the cluster. As the foregoing illustrates, however, a problem arises in that administrative requests are handled differently in the Tuxedo and MSCS environments. In the Tuxedo architecture, administrative requests, including bringing a component online or offline, are performed by a designated master node. On the contrary, in the MSCS environment, administrative requests are performed by the resource DLL on the node to which the operation is directed, not by a designated master node. The present invention provides a solution to this problem.